Chapter 3: Linear Regression¶
Linear Regression
The linear regression model is expressed as follows
\begin{align}
Y= \beta_0 + \sum_{j=1}^{p}\beta_{j} X_{j},
\end{align}
where $X_j$ represents the j-th predictor and $\beta_j$ quantifies the association between that variable and the response.
In case that $p=2$, $\beta_0$ and $\beta_1$ are known the intercept and slope terms, respectively.
In practice, we cannot identify $\beta_0,~\beta_1,~\ldots,~\beta_p$. Instead, we can have estimates $\hat{\beta}_0,\hat{\beta}_1, \ldots, \hat{\beta}_p$. Given estimates $\hat{\beta}_0$, $\hat{\beta}_1$, $\ldots$, $\hat{\beta}_p$, the following formula can be used for predictions,
\begin{align}
\hat{y}=\hat{\beta}_0 +\hat{\beta}_1x_1 +\hat{\beta}_2 x_2 + \ldots +\hat{\beta}_p x_p.
\end{align}
Using the least-squares approach, $\beta_0$, $\beta_1$, $\ldots$, $\beta_p$ can be chosen to minimize the sum of squared residuals
\begin{align}
RSS=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2=\sum_{i=1}^{n}(y_i-\hat{\beta}_0 - \hat{\beta}_1 x_{i1} - \hat{\beta}_2 x_{i2} - \hat{\beta}_3 x_{i3}-\ldots - \hat{\beta}_p x_{ip})^2.
\end{align}
There are a few definitions that we need throughout this document.
$R^2$ Statistic¶
The $R^2$ statistic is a measure of the linear relationship between $X$ and $Y$. To calculate $R^2$ , we use the formula
\begin{align}
R^2 =1-\frac{\sum_{i=1}^{n}(y_i − \hat{y}_i) 2}{\sum_{i=1}^{n}(y_i − \bar{y}) 2}
\end{align}
Correlation Matrix¶
It is a matrix in which i-j position defines the correlation between the ith and jth parameter of the given data-set. Defined as correlation
\begin{align}
Cor(X,Y )=\frac{\sum_{i=1}^{n}(x_i − \bar{x})(y_i − \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i − \bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i − \bar{y})^2}}
\end{align}
Correlation Matrix is basically a covariance matrix. Also known as the auto-covariance matrix, dispersion matrix, variance matrix, or variance-covariance matrix.
